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DETAILED ACTION 



1. This action is responsive to communications: Appeal Brief filed 21 Feb. 2007. 

2. Claims 1-28 are pending in this case. Claims 1, 16 and 22 are independent 
claims. 



3. In view of the Appeal Brief filed on 21 Feb. 2007, PROSECUTION IS HEREBY 
REOPENED. A new grounds of rejection is set forth below. 

To avoid abandonment of the application, appellant must exercise one of the 
following two options: 

(1 ) file a reply under 37 CFR 1.111 (if this Office action is non-final) or a reply 
under 37 CFR 1.113 (if this Office action is final); or, 

(2) initiate a new appeal by filing a notice of appeal under 37 CFR 41 .31 followed 
by an appeal brief under 37 CFR 41 .37. The previously paid notice of appeal fee and 
appeal brief fee can be applied to the new appeal. If, however, the appeal fees set forth 
in 37 CFR 41.20 have been increased since they were previously paid, then appellant 
must pay the difference between the increased fees and the amount previously paid. 

A Supervisory Patent Examiner (SPE) has approved of reopening prosecution by 
signing below: 



Reopening of Prosecution After Appeal Brief or Reply Brief 




Technology Center 2100 
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Claim Rejections - 35 USC §112 

4. The following is a quotation of the first paragraph of 35 U.S. C. 112: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

5. Claims 3 and 18 are rejected under 35 U.S.C. 112, first paragraph, as failing to 
comply with the enablement requirement. The claim(s) contains subject matter "explicit 
models", which was not described in the specification in such a way as to enable one 
skilled in the art to which it pertains, or with which it is most nearly connected, to make 
and/or use the invention. 

6. Claims 4 and 19 are rejected under 35 U.S.C. 112, first paragraph, as failing to 
comply with the enablement requirement. The claim(s) contains subject matter "implicit 
models", which was not described in the specification in such a way as to enable one 
skilled in the art to which it pertains, or with which it is most nearly connected, to make 
and/or use the invention. 

Claim Rejections - 35 USC §112 

7. The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 
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8. Claims 3 and 18 are rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. The claim recite "deterministically or probabilistically 
is based on explicit models" in which the Examiner determines to be indefinite in this 
context as the specification offers no clarity as to it's meaning. 

9. Claims 4 and 19 are rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. The claim recite "deterministically or probabilistically 
is based on implicit models" in which the Examiner determines to be indefinite in this 
context as the specification offers no clarity as to it's meaning. 
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Claim Rejections - 35 USC § 112 

10. Claims 14 and 15 recites the limitation "the method of claim 2, wherein the 

models " . Claim 2 does not recite "models". There is insufficient antecedent basis 

for this limitation in the claim. The Examiner concludes Applicant meant for claims to be 
dependency of claim 13 and will be examined accordingly. 

Claim Rejections - 35 USC § 103 

1 1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

12. Claims 1, 6, 9, 10, 12, 16, 21, 22 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Stubler et al. (Patent No.: US 6,804,684, B2; Filed May 7, 2001) 
(hereinafter "Stubler") in view of LeBrun et al. (Patent No.: 6,043,819; Filed Jul. 01, 
1997) (hereinafter "LeBrun). 

In regards to independent claims 1, 16 and 22, Stubler discloses method for 
generating persistent annotations of multimedia content, comprising one or more 
repetitions of the following steps: 

actively selecting examples of multimedia content to be annotated by a user (fig. 
2, fig. 6-7, col. 3 line 46 - col. 4 line 12, col. 8 lines 18-23, and col. 9 line 65 - col. 10 
line 18; col. 11, lines 3-35; Stubler discloses unlabeled image regions being presented 
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to the user so that the user can apply a caption or label to all of the regions 
simultaneously. Stubler also disclose automatically generating one or more captions or 
labels for the acquired image automatically without any user intervention.). 

accepting input annotations from said user for said selected examples (fig. 2, col. 
3 line 46 -col. 4 line 12, and col. 8 lines 18-55; Stubler discloses unlabeled image 
regions being presented to the user so that the user can apply a caption or label to all of 
the regions simultaneously.). 

propagating said input annotations to other instances of multimedia content (fig. 
2; col. 3 line 46 - col. 4 line 12; col. 8 lines 18-55; col. 2 line 59 - col. 3 line 10; Stubler 
discloses unlabeled image regions being presented to the user so that the user can 
apply a caption or label to all of the regions simultaneously.). 

storing said input annotations and said propagated annotations (col. 8 lines 1 8- 

55). 

Stubler does not expressly disclose actively selecting examples of multimedia 
content to be annotated by a user, wherein the examples of multimedia content are 
selected based on at least one criterion for achieving a maximal disambiguation result 
such that only those examples which are most ambiguous are selected. 

However LeBrun teaches actively selecting examples of multimedia content to be 
annotated by a user, wherein the examples of multimedia content are selected based 
on at least one criterion for achieving a maximal disambiguation result such that only 
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those examples which are most ambiguous are selected (col. 3, lines 49-51; col. 10, 
lines 4-8; col. 18, lines 16-20; col. 21, lines 20-24; LeBrun teaches a database of 
graphic document images which are automatically identified (annotated) by an image 
character reader. Images not automatically identified (annotated) by the image 
character reader are queued for manual identification/classification by human operators 
looking sequentially at a screen or queue of images and keying in the identification.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler with LeBrun for the benefit of speeding up the 
process of document (multimedia annotations) processing so that a higher volume of 
transactions can be processed, and reduce the number of errors which are considered 
to be inherent in a document processing (multimedia annotations) operation (col. 3, 
lines 14-23). 

In regards to dependent claims 6 and 21, Stubler discloses wherein the 
multimedia content comprises one or more types selected from the group consisting of: 
images, audio, video, graphics, text, multimedia, Web pages, time series data, 
surveillance data, sensor data, relational data, and XML data (col. 3 line 46 - col. 4 line 
12; Stubler discloses images type multimedia.). 

In regards to dependent claim 9, Stubler discloses the method of claim 1, 
wherein the process of creating input annotations by the user involves multimodal 
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interaction with the user using graphical, textual, and/or speech interface (col. 9, line 65- 
col. 10, line 18; Stubler disclose an interactive user verification stage in which the user 
may select and/or edit captions and label.). 

In regards to dependent claim 10, Stubler discloses the method of claim 1, 
wherein the input annotations are created by means of steps selected from the group 
consisting of: creating new annotations, deleting existing annotations, rejecting 
proposed annotations, and modifying annotations (col. 9, line 65-col. 10, line 18; Stubler 
disclose an interactive user verification stage in which the user may select and/or edit 
captions and label.). 

In regards to dependent claim 12, Stubler discloses the method of claim 9, 
wherein the multimodal interaction involves speech recognition, gaze detection, finger 
pointing, expression detection, and/or effective computing methods for sensing a user's 
state (col. 5, lines 59-63; col. 6, line 55-62; col. 9 line 49 - col. 10 line 18). 

Note 

13. It is noted that any citations to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the reference should not be considered to 
be limiting in any way. A reference is relevant for all it contains and may be relied upon 
for all that it would have reasonably suggested to one having ordinary skill in the art. 
See MPEP 2123. 
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14. Claims 2-5 and 17-20 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Stubler in view of LeBrun further in view of Lennon et al. 
(Patent No.: 6,718,063 B1; Filed Dec. 10, 1999) (hereinafter "Lennon"). 

In regards to dependent claims 2 and 17, Stubler discloses wherein the step of 
actively selecting is performed using a selection technique selected from deterministic 
(col. 4 line 64 - col. 5 line 19). 

Stubler in view of LeBrun does not expressly disclose actively selecting is 
performed using a selection technique of probabilistic. 

However Lennon teaches actively selecting is performed using a selection 
technique of probabilistic (col. 3, lines 19-35; Lennon teaches using a probabilistic 
method for correctly assigning labels to regions of images.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Lennon for the benefit of 
using a probabilistic method, which results in each assigned label for a region having an 
associated probability or likelihood of the label being correctly assigned (col. 3, lines 23- 
27). 

In regards to dependent claims 3 and 18, Stubler discloses wherein the step of 
actively selecting, which is performed deterministically or probabilistically, is based on 
explicit models and feature proximity/similarity measures, and returns one or more 
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examples of multimedia content to be annotated (fig. 2 and col. 8 lines 18-55). 

In regards to dependent claims 4 and 19, Stubler discloses wherein the step of 
actively selecting, which is performed deterministically or probabilistically, is based on 
implicit models and feature proximity/similarity measures, and returns one or more 
examples of multimedia content to be annotated (fig. 2 and col. 8 lines 18-55). 

In regards to dependent claims 5 and 20, Stubler in view of LeBrun does not 
expressly disclose wherein an optimization criterion for active selection includes one or 
more criteria selected from the group consisting of; information measures and 
confidence. 

However Lennon teaches wherein an optimization criterion for active selection 
includes one or more criteria selected from the group consisting of: information 
measures and confidence (Abstract; col. 7, lines 41-45; Lennon teaches a probability 
value expressing the confidence level of the label being correctly assigned.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Lennon for the benefit of 
using a probabilistic method, which results in each assigned label for a region having an 
associated probability or likelihood of the label being correctly assigned (col. 3, lines 23- 
27). 

Note 
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15. It is noted that any citations to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the reference should not be considered to 
be limiting in any way. A reference is relevant for all it contains and may be relied upon 
for all that it would have reasonably suggested to one having ordinary skill in the art. 
See MPEP2123. 

16. Claims 7, 8, 11, and 13-15 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Stubler in view of LeBrun further in view of Lipson et al. 
(Patent No.: 5,963,670; Filed: Feb. 12, 1996) (hereinafter "Lipson"). 

In regards to dependent claim 7, Stubler in view of LeBrun does not expressly 
disclose the method of claim I, wherein the input annotations are created by a user with 
reference to a vocabulary. 

However Lipson teaches the method of claim /, wherein the input annotations are 
created by a user with reference to a vocabulary (col. 9, lines 23-35; Lipson teaches a 
rich vocabulary to differentiate between many classes of images.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Lipson for the benefit of 
providing a class model, which can be used to detect images of that class in a database 
(col. 2, lines 39-41). 
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In regards to dependent claim 8, Stubler in view of LeBrun does not expressly 
disclose the method of claim 7, wherein the vocabulary contains one or more items 
selected from the group consisting of: terms, concepts, labels, and annotations. 

However Lipson teaches the method of claim 7, wherein the vocabulary contains 
one or more items selected from the group consisting of: terms, concepts, labels, and 
annotations (col. 9, lines 23-42; Lipson teaches a rich vocabulary to differentiate 
between many classes of images. It has been established and is commonly known that 
vocabularies typically contain terms and concepts.). • 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Lipson for the benefit of 
providing a class model, which can be used to detect images of that class in a database 
(col. 2, lines 39-41). 

In regards to dependent claim 11, Stubler in view of LeBrun does not expressly 
disclose the method of claim 7, wherein the vocabulary is adaptively or dynamically 
organized and/or limited by the system or the user. 

However Lipson teaches the method of claim 7, wherein the vocabulary is 
adaptively or dynamically organized and/or limited by the system or the user (col. 9, 
lines 23-35.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Lipson for the benefit of 
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providing a class model, which can be used to detect images of that class in a database 
(col. 2, lines 39-41). 

In regards to dependent claim 13, Stubler discloses the method of claim 1, 
wherein the determination of the propagation of annotations is made deterministically or 
probabilistically (col. 4 line 64 - col. 5 line 19) 

Stubler in view of LeBrun does not expressly disclose the use of models for each 
annotation or for joint annotations. 

However Upson teaches the use of models for each annotation or for joint 
annotations (col. 2, lines 56-59; Lipson teaches an image model for classifying or 
detecting images.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Lipson for the benefit of 
providing a class model, which can be used to detect images of that class in a database 
(col. 2, lines 39-41). 

In regards to dependent claim 14, Stubler in view of LeBrun does not expressly 
disclose the method of claim 2, wherein the models are created or learned automatically 
or sem'hautomatically and/or are updated adaptively from interaction with the user. 

However Lipson teaches wherein the models are created or learned 
automatically or semi-automatically and/or are updated adaptively from interaction with 
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the user (col. 9, line 37-col. 10, line 22; Lipson teaches a method for generating a class 
model.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Lipson for the benefit of 
providing a class model, which can be used to detect images of that class in a database 
(col. 2, lines 39-41). 

In regards to dependent claim 15, Stubler in view of LeBrun does not expressly 
disclose the method of claim 2, wherein the models are based on nearest neighbor 
voting or variants, parametric or statistical models, expert systems, rule-based systems, 
or hybrid techniques. 

However Lipson teaches the use of models for each annotation or for joint 
annotations (col. 2, lines 56-59; col. 13, line 2-col. 14, Iine18). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Lipson for the benefit of 
providing a class model, which can be used to detect images of that class in a database 
(col. 2, lines 39-41). 

Note 

17. It is noted that any citations to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the reference should not be considered to 
be limiting in any way. A reference is relevant for all it contains and may be relied upon 
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for all that it would have reasonably suggested to one having ordinary skill in the art. 
See MPEP 2123. 

18. Claims 23, 25 and 27 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Stubler and LeBrun in view of Neal et al. (Patent No.: US 
6,697,799 B1; Effective Filling Date: Sep. 10, 1999) (hereinafter "Neal"). 

In regards to dependent claims 23, 25 and 27, Stubler in view of LeBrun does 
not expressly disclose wherein the at least one criterion includes an ambiguity level of 
the selected examples. 

However Neal teaches wherein the at least one criterion includes an ambiguity 
level of the selected examples (col. 11, lines 1-47; Neal teaches a classification 
confidence score which determines the level of confidence in which a category is likely 
to be correct during classification. If the item has a high confidence, then it can be 
classified directly. If the confidence level is low, then the results can be sent to the user 
interface for review and selection by the operator. Using the broadest reasonable 
interpretation, the Examiner has determined that the "confidence level" as taught by 
Neal is analogous with the "ambiguity level" of the current invention.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Neal for the benefit of 
comparing confidence scores of items for all selected classifications and classifying the 
items based on confidence score comparison (col. 2, lines 36-42). 
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Note 

19. It is noted that any citations to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the reference should not be considered to 
be limiting in any way. A reference is relevant for all it contains and may be relied upon 
for all that it would have reasonably suggested to one having ordinary skill in the art. 
See MPEP 2123. 

20. Claims 24, 26 and 28 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Stubler and LeBrun in view of Neal, further in view of Toyama 
(Patent No.: US 6,816,847 B1; Filling Date: Sep. 23, 1999). 

In regards to dependent claims 24, 26 and 28, Stubler in view of LeBrun does 
not expressly disclose wherein the at least one criterion includes a confidence level of 
the selected examples, the confidence level being inversely proportional to a distance of 
a new feature of the selected examples from a separating hyperplane in an induced 
higher dimensional feature space. 

Neal teaches wherein the at least one criterion includes an ambiguity level of the 
selected examples (col. 11, lines 1-47; Neal teaches a classification confidence score 
which determines the level of confidence in which a category is likely to be correct 
during classification. If the item has a high confidence, then it can be classified directly. 
If the confidence level is low the results can be sent to the user interface for review and 
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selection by the operator. Using the broadest reasonable interpretation, the Examiner 
has determined that the "confidence level" as taught by Neal is analogous with the 
"ambiguity level" of the current invention.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Neal for the benefit of 
comparing confidence scores of items for all selected classifications and classifying the 
items based on confidence score comparison (col. 2, lines 36-42). 

Toyama teaches a distance of a new feature of the selected examples from a 
separating hyperplane in an induced higher dimensional feature space (col. 5, line 47- 
col. 6, line 50; Toyama teaches a SVM classifier by identifying a hyperplane that 
separates a set of positive and negatives examples with a maximum margin. Toyama 
further teaches, the quality of the learned classifiers for aesthetic image judgment can 
be enhanced by inputting to the feature selection procedures that are useful for 
distinguishing different aesthetic among images. Using the broadest interpretation, the 
Examiner concludes at the time of the invention, one of ordinary skill of the art could 
modify Neal's teaching with Toyama teaching of enhancing the learned qualifier in such 
a way that wherein the at least one criterion includes a confidence level of the selected 
examples, the confidence level being inversely proportional to a distance of a new 
feature of the selected examples from a separating hyperplane in an induced higher 
dimensional feature space.). 
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Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler, LeBrun and Neal with Toyama for the benefit 
of generating a set of images for input into a SVM classifier (col. 1 , lines 57-64). 

Note 

21 . It is noted that any citations to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the reference should not be considered to 
be limiting in any way. A reference is relevant for all it contains and may be relied upon 
for all that it would have reasonably suggested to one having ordinary skill in the art. 
See MPEP 2123. 

Response to Arguments 

22. Applicant's arguments, see Appeal Brief, filed 21 Feb. 2007, have been fully 
considered and are persuasive. Therefore, the rejection has been withdrawn. 
However, upon further consideration, a new ground(s) of rejection is made in view of 
Stubler, LeBrun, Lennon, Lipson, Neal and Toyama. 
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Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to James J. Debrow whose telephone number is 571-272- 
5768. The examiner can normally be reached on 8:00-5:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Heather Herndon can be reached on 571-272-4136. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

JAMES DEBROW 
EXAMINER 
ART UNIT 2176 
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